Skip to content

numerics: saturate the prompt path's FP16 SwiGLU products; log1pf in the unfused GDN gate - #281

Closed
sergqwer wants to merge 1 commit into
Niko1221:mainfrom
sergqwer:pr/numerics
Closed

numerics: saturate the prompt path's FP16 SwiGLU products; log1pf in the unfused GDN gate#281
sergqwer wants to merge 1 commit into
Niko1221:mainfrom
sergqwer:pr/numerics

Conversation

@sergqwer

@sergqwer sergqwer commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Two numerical edge cases found in a read-only audit of the prompt path:

  • FP16 overflow in SwiGLU. The products that feed the prompt path's FP16 down projection are now clamped to
    FP16's range (±65504) before rounding. A token with a massive activation otherwise becomes inf there, and NaN
    after the GEMM. Decode's q8_1 path (range ~8e6) stays finite for the same token.
  • Softplus underflow. The unfused GDN gate used logf(1 + expf(v)), which returns 0 for v below ~-16 (1 + e^v
    rounds to 1). The fused kernels, gdn.cu and verify_kernels.cu already use log1pf(expf(v)); now the unfused
    path does too.

Measured

Measured on current main (0.1.29, d6708a4) with ISTA-DASLab's IQ2_XS, RTX 5090 32 GB, Ryzen 9 9950X3D, 128 GB DDR5, Windows 11. KL is first-token KL divergence (STRATA_DUMP_FIRST_LOGITS, #276); two runs of main itself differ by 0 on 5 of 6 short prompts and by 0.00013 on the sixth. On ordinary prompts nothing moves: first-token KL to main 0, 0.00013 and 0 on three prompts
(1K, 2K and 4K tokens), which is the run-to-run noise itself.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

sergiywith @claude
numerics: saturate the prompt path's FP16 SwiGLU products; log1pf in …
a3af1ed 
…the unfused GDN gate

- prefill/kernels.cu: the SwiGLU products that feed the prompt path's FP16 down projection are clamped to FP16's
  range (+-65504) before rounding. A token with a massive activation otherwise becomes inf there and NaN after the
  GEMM, where decode's q8_1 path (range ~8e6) stays finite.
- native_gdn_preprocess.cu: the unfused gate softplus used logf(1 + expf(v)), which returns 0 for v below ~-16
  (1 + e^v rounds to 1); the fused kernels, gdn.cu and verify_kernels.cu already use log1pf(expf(v)). The two paths
  now agree.

On ordinary prompts nothing moves: first-token KL to main 0, 0.00013, 0 (the run-to-run noise).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio
@Niko1221

Niko1221 commented Oct 1, 2026

Copy link
Copy Markdown
Owner

Thanks! This is in the 0.1.32 batch. We take it as is (we may keep NaN propagating through the clamp in a follow-up); every change goes through our identity gate (byte-identical output on four models) and a speed A/B before it ships.

Niko1221 added a commit that referenced this pull request Oct 1, 2026
@Niko1221 @claude
#281 follow-up: the saturated FP16 SwiGLU product keeps a NaN a NaN
f8fe938 
fmaxf(NaN, -65504) is -65504, so #281's clamp turned a NaN product into -65504 and the prompt path went on with a
plausible value where it used to show the NaN (STRATA_DBG_NAN and the NaN checks find it there). isnan(x) ? x :
clamp(x): only +-inf (and finite values from 65520 up, which rounded to inf) saturate; everything else is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
@Niko1221

Niko1221 commented Oct 1, 2026

Copy link
Copy Markdown
Owner

Released in 0.1.32 with NaN kept as NaN through the clamp. Merged under your name (GitHub doesn't show it as merged because it went in by hand). Thank you!

@Niko1221 Niko1221 closed this Oct 1, 2026
@rwkeyes rwkeyes mentioned this pull request Oct 4, 2026
Closed
Niko1221 added a commit that referenced this pull request Oct 6, 2026
@Niko1221
#281 follow-up: the saturated FP16 SwiGLU product keeps a NaN a NaN
6280f65 
fmaxf(NaN, -65504) is -65504, so #281's clamp turned a NaN product into -65504 and the prompt path went on with a
plausible value where it used to show the NaN (STRATA_DBG_NAN and the NaN checks find it there). isnan(x) ? x :
clamp(x): only +-inf (and finite values from 65520 up, which rounded to inf) saturate; everything else is unchanged.
anon761 pushed a commit to anon761/Strata-upstream that referenced this pull request Oct 6, 2026
@Niko1221
Niko1221#281 follow-up: the saturated FP16 SwiGLU product keeps a NaN…
6716123 
… a NaN

fmaxf(NaN, -65504) is -65504, so Niko1221#281's clamp turned a NaN product into -65504 and the prompt path went on with a
plausible value where it used to show the NaN (STRATA_DBG_NAN and the NaN checks find it there). isnan(x) ? x :
clamp(x): only +-inf (and finite values from 65520 up, which rounded to inf) saturate; everything else is unchanged.
anon761 pushed a commit to anon761/Strata-upstream that referenced this pull request Oct 6, 2026
@Niko1221 @claude
Niko1221#281 follow-up: the saturated FP16 SwiGLU product keeps a NaN…
dbbb6f6 
… a NaN

fmaxf(NaN, -65504) is -65504, so Niko1221#281's clamp turned a NaN product into -65504 and the prompt path went on with a
plausible value where it used to show the NaN (STRATA_DBG_NAN and the NaN checks find it there). isnan(x) ? x :
clamp(x): only +-inf (and finite values from 65520 up, which rounded to inf) saturate; everything else is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
k0valik added a commit to k0valik/Strata that referenced this pull request Oct 15, 2026
@k0valik
replay(githubproxy.fjygbaifeng.eu.org-eelgaev-Strata-AC922-ac922): src/prefill from ac922
69598f0 
Squashed from 920 commit(s) fb58e0d..1f029aa on https://github.com/eelgaev/Strata-AC922, restricted to the
paths this fork still mirrors: 3 file(s) under src/prefill -- 2 clean,
1 with conflict markers.

Conflict-marked files -- a pull request with conflicts: resolve them,
do not discard the work:
  src/prefill/prefill.cpp

Original authors: Andy,Anon,BlueKingMuch,Chen Bao

Original commits touching these paths (123 subject(s)):
  - Niko1221#136 P3: with the fused experts the prompt path's MoE buffers hold only what the fused kernels need
  - Niko1221#136: fused int8 tensor-core experts for the Q2_0 pack's prompt path, grouped on the GPU (opt-in STRATA_PF_FUSED=1)
  - Niko1221#136: the fused experts take a 1024-slot streamed ring by default (two layers' experts)
  - Niko1221#136: the fused experts take a 512-slot streamed ring (a whole layer's experts) by default
  - Niko1221#136: the fused int8 prompt experts for the native IQ packs (gate/up IQ2_XXS IQ2_XS IQ2_S IQ3_XXS IQ3_S IQ4_XS, down Q2_0 IQ4_NL; opt-in STRATA_PF_FUSED=1)
  - Niko1221#253: the 8 GiB arena pin cap only under WDDM; STRATA_ARENA_PIN_GIB; the split's prompt chunk steps down
  - Niko1221#280 follow-up: the rope kernels without the table are 0.1.31's code exactly (template <bool TAB>)
  - Niko1221#281 follow-up: the saturated FP16 SwiGLU product keeps a NaN a NaN

Review artifact produced by fork-replay.sh (a spike adapted from
sync-upstream.sh): a filtered, squashed replay for reading -- not built,
not tested, and not evidence that any of it works.

Fork-Replay-Source: https://github.com/eelgaev/Strata-AC922
Fork-Replay-Branch: ac922
Fork-Replay-Base: fb58e0d
Fork-Replay-Tip: 1f029aa
Fork-Replay-Group: src/prefill
Fork-Replay-Automated: true
k0valik added a commit to k0valik/Strata that referenced this pull request Oct 15, 2026
@k0valik
replay(githubproxy.fjygbaifeng.eu.org-eelgaev-Strata-AC922-ac922): src/prefill from ac922
e9c0181 
Squashed from 920 commit(s) fb58e0d..1f029aa on https://github.com/eelgaev/Strata-AC922, restricted to the
paths this fork still mirrors: 3 file(s) under src/prefill -- 2 clean,
1 with conflict markers.

Conflict-marked files -- a pull request with conflicts: resolve them,
do not discard the work:
  src/prefill/prefill.cpp

Original authors: Andy,Anon,BlueKingMuch,Chen Bao

Original commits touching these paths (123 subject(s)):
  - Niko1221#136 P3: with the fused experts the prompt path's MoE buffers hold only what the fused kernels need
  - Niko1221#136: fused int8 tensor-core experts for the Q2_0 pack's prompt path, grouped on the GPU (opt-in STRATA_PF_FUSED=1)
  - Niko1221#136: the fused experts take a 1024-slot streamed ring by default (two layers' experts)
  - Niko1221#136: the fused experts take a 512-slot streamed ring (a whole layer's experts) by default
  - Niko1221#136: the fused int8 prompt experts for the native IQ packs (gate/up IQ2_XXS IQ2_XS IQ2_S IQ3_XXS IQ3_S IQ4_XS, down Q2_0 IQ4_NL; opt-in STRATA_PF_FUSED=1)
  - Niko1221#253: the 8 GiB arena pin cap only under WDDM; STRATA_ARENA_PIN_GIB; the split's prompt chunk steps down
  - Niko1221#280 follow-up: the rope kernels without the table are 0.1.31's code exactly (template <bool TAB>)
  - Niko1221#281 follow-up: the saturated FP16 SwiGLU product keeps a NaN a NaN

Review artifact produced by fork-replay.sh (a spike adapted from
sync-upstream.sh): a filtered, squashed replay for reading -- not built,
not tested, and not evidence that any of it works.

Fork-Replay-Source: https://github.com/eelgaev/Strata-AC922
Fork-Replay-Branch: ac922
Fork-Replay-Base: fb58e0d
Fork-Replay-Tip: 1f029aa
Fork-Replay-Group: src/prefill
Fork-Replay-Automated: true
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sergqwer @Niko1221